Skip to content

feat: compress complex all null #4990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
yingjianwu98 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: compress complex all null #4990

yingjianwu98 wants to merge 1 commit into from
+288 −27

Conversation

@yingjianwu98
Copy link
Contributor

@yingjianwu98 yingjianwu98 commented Oct 17, 2025 •
edited
Loading

In order to compress complex all null, we need to add additional parameters in the proto so we know what compression are used for definition level and repetition level and the number of values accordingly.

resolve #4885

@github-actions github-actions bot added the enhancement New feature or request label Oct 17, 2025
@yingjianwu98
Copy link
Contributor Author

yingjianwu98 commented Oct 17, 2025 •
edited
Loading

Have this initial implementation ready for review, and if this is on the right track, I will work on changes to make it backward compatible.

@yingjianwu98 yingjianwu98 force-pushed the compressComplexAllNull branch from 1238940 to 2cdd916 Compare October 17, 2025 21:27
@yingjianwu98 yingjianwu98 mentioned this pull request Oct 17, 2025
Open
@codecov-commenter
Copy link

codecov-commenter commented Oct 17, 2025

Codecov Report

❌ Patch coverage is 86.22449% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.74%. Comparing base (cd4a91e) to head (2cdd916).
⚠️ Report is 22 commits behind head on main.

Files with missing lines Patch % Lines
.../lance-encoding/src/encodings/logical/primitive.rs 85.32% 15 Missing and 12 partials ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #4990      +/-   ##
==========================================
+ Coverage   81.72%   81.74%   +0.01%     
==========================================
  Files         334      340       +6     
  Lines      133358   137163    +3805     
  Branches   133358   137163    +3805     
==========================================
+ Hits       108986   112119    +3133     
- Misses      20720    21312     +592     
- Partials     3652     3732      +80
Flag Coverage Δ
unittests 81.74% <86.22%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Xuanwo
Copy link
Collaborator

Xuanwo commented Jan 14, 2026

Thank you for working on this, and sorry for the late review. I believe this PR can continue the work from #5641. Are you still interested in working on it? If you don't have time, I can continue the work base on this PR.

@yingjianwu98
Copy link
Contributor Author

yingjianwu98 commented Jan 14, 2026

@Xuanwo

Thanks for the review!
I don't want to block your development but If this is not too urgent, I can work on a PR for review this this week if that's ok.

@Xuanwo
Copy link
Collaborator

Xuanwo commented Jan 28, 2026

@Xuanwo

Thanks for the review! I don't want to block your development but If this is not too urgent, I can work on a PR for review this this week if that's ok.

@yingjianwu98 Appreciated! #5641 was merged today. Do you have time to rebase your work on top of it? Thank you in advance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Rep/Def levels should be compressed in the all null page layout

3 participants

@yingjianwu98 @codecov-commenter @Xuanwo